docs(rfc): consumer gate — runtime stop/start of individual queue controllers#374
docs(rfc): consumer gate — runtime stop/start of individual queue controllers#374sbalabanov wants to merge 1 commit into
Conversation
…trollers Adds the Consumer Gate RFC and links it from the RFC index. The RFC proposes a consumer middleware that parks deliveries in-flight (with periodic visibility extension, so no retry burn or DLQ) while a controller's gate is closed. Gate state is a separate extension at platform/extension/consumergate with a file-based first implementation, giving e2e tests and operators a stop/observe/start primitive for individual queue controllers without stopping the hosting service. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
|
|
||
| The gate is a decorator installed by the consumer around every registered controller. Before invoking `Process`, it consults gate state for the controller's consumer group. If the gate is closed, the delivery is **parked**: the decorator blocks in place, keeping the delivery in-flight and periodically calling `ExtendVisibilityTimeout` — already part of the `Delivery` contract, and specified to *not* increment the retry count — until the gate opens or the consumer shuts down. When the gate opens, the parked delivery proceeds into the controller as the same attempt, in partition order. Parked messages therefore never burn retry budget, never touch the DLQ, and are never lost: if the process dies while parked, extension stops, visibility lapses, and the queue redelivers normally. | ||
|
|
||
| Stopping is a barrier, not preemption: a message already inside `Process` when the gate closes runs to completion; the gate guarantees no *new* message enters the controller. |
There was a problem hiding this comment.
the one other i have noticed which is not testing from functional perspective we need, when we have limits on how much a stage can do and requires queue to be halted..but that signal is generally coming from Process itself on consumption of the message, like merge serialization... so yeah, extending visibility but don't account to retry (explicit) would be great where we consume the message and return with some nack+visibilityTimeout, i thought about adding it when i added PublishAfter semantic for build status polling but as I am doing the merge/finalize stuff..seems like we are missing this construct.. so I am hoping we can use the same there as well
|
|
||
| Stopping is a barrier, not preemption: a message already inside `Process` when the gate closes runs to completion; the gate guarantees no *new* message enters the controller. | ||
|
|
||
| One bounded side effect is accepted and documented: the routing loop feeds each partition through a channel buffered at the subscription's batch size, so if messages keep arriving for a parked partition, the topic's routing loop eventually stalls once that buffer fills. For a fully closed gate this is moot (every partition parks anyway), and at test volumes it never triggers. |
There was a problem hiding this comment.
i think even for actual volume, it comes down to buffer size and other wise everything remain in DB i guess for later delivery.
|
|
||
| The first implementation stores gate state as plain files under a configured directory. Presence of a gate file means the gate is closed; deleting the file opens it. Parked deliveries are recorded as JSON files. The layout: | ||
|
|
||
| ``` |
There was a problem hiding this comment.
if we want to use it for backoff business logic, we may need to store them in DB
4 participants
This RFC proposes a first-class stop/observe/start primitive for individual queue controllers: halt exactly one controller from taking new messages, observe some condition while it is stopped, then start it again — without stopping the service that hosts it. Motivations are deterministic e2e scenario control (interleaving in-flight messages across controllers) and operational pause of a consuming stage during incidents.
The mechanism is consumer middleware: a decorator installed around every registered controller consults gate state before invoking
Process. When the gate is closed, the delivery is parked in-flight — the decorator blocks and periodically extends the visibility timeout, so parked messages never burn retry budget, never hit the DLQ, and are redelivered normally if the process dies. When the gate opens, the delivery proceeds as the same attempt, in partition order.Gate state lives in a new shared extension at
platform/extension/consumergate, with a file-based first implementation usable by both e2e tests and operators. This is a design doc only — no code changes.🤖 Generated with Claude Code